A System for Multimodal Dialogue and Language Acquisition

Human-computer interaction can be enhanced by employing multiple modalities in the dimensions of sound, sight, and touch. The integration of speech with gestures, can make the human-computer dialogue more natural and efficient. However, even for limited domain applications, computer dialogue systems impose a constraint upon the language rules and vocabulary that users can employ. This is because of the difficulty of accounting for all the surface linguistic forms that end-users might utilize. In this paper we present a system for multimodal dialogue capable of acquiring new vocabulary items, at run-time, from end-users. The system contains a multimodal interface based on speech, typing, pointing, drawing, mouse clicking and image capturing. Information from different input modalities is combined to interpret user’s intent and solve ambiguities from incomplete, anaphoric, or elliptical utterances such as, ‘Move this to there’ and ‘Now here’. The system can acquire, through a set of learning rules, new vocabulary items and the related semantics. The grammar can thus be adapted and personalized by the end-users according to their needs and preferences.